Image generating apparatus and image generating method

ABSTRACT

To generate motion picture image data making full use of the characteristics of a photographed image such as a high resolution and a high rate fetching (high frame rate), an image pickup apparatus comprises a camera unit for photographing a subject at predetermined time intervals and generating image pickup data and an encoding unit for encoding the image pickup data as motion picture image data having an image rate which is based on the predetermined time intervals. An optimum motion picture stream is output by changing the output mode of the encoded motion picture image data output from the encoding unit according to the number of photographing pixels in the camera unit and the image rate.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technology of an image generatingapparatus and an image generating method for fetching a motion pictureat a variable frame rate and generating compression-encoded image data.

2. Related Background Art

There are digital still cameras and digital video cameras as devicescapable of photographing an image and recording and reproducing theimage. Recent image pickup elements employ an increased number of pixelsdue to an advanced manufacturing technology, thereby the quality ofimages photographed by not only digital still cameras but also bydigital video cameras using the image pickup elements is greatlyimproved (resolution of the images is increased).

When an image having a high resolution is generated using the camerasdescribed above, although the photographing capability, imagecompression capability, recording capability, and the like of them aregreatly influenced due to a large amount of image data, the operatingconditions of the cameras are optimally set according to the respectivecameras and the modes thereof by restricting a processing time and anamount of processing data.

For example, there is a digital video camera which records pickup dataof 480×720 pixels in NTSC or 576×720 pixels in PAL (corresponding to640×480 square pixels in VGA) at a rate of 30 frames/sec in a motionpicture recording mode which assumes that a motion picture is output toa TV monitor, and which records pickup data of 1,230,000 pixels(corresponding to square pixels 1280×960 pixels in SXVGA) in a stillimage recording mode.

Further, there is a popular type digital still camera which records astill image of 3,000,000 pixels (corresponding to 2048×1536 pixels inQXGA) or more and also continuously photographs a motion picture at arate of about 2 frames/sec.

In general, however, even a digital video camera and a digital stillcamera having an image pickup element including a large number of pixelscan only photograph and record an image in a fixed pixel range that isuniquely determined according to a photographing mode and an image sizeset by a user, and it is difficult to optionally obtain a photographedimage having a high resolution of, in particular, a motion picture.

In contrast, for the digital video cameras, users have a desire forexecuting photographing at a high rate, in addition to a desire forincreasing the number of pixels. For example, there is the need forchanging a frame rate to a high rate side even while a motion picture isbeing photographed. However, the increase in the frame rate not onlyincreases a load on respective processing circuits but also causes apossibility that a reproduction unit or a display unit cannot cope witha motion picture photographed at a changed frame rate when the pictureis reproduced. Thus, these problems must be overcome when a digitalvideo camera that fulfills the need is actually manufactured as aproduct. In view of the above problems, there is conceived a motionpicture photographing system that makes it possible to record a motionpicture at a different frame rate by controlling the system by a hostcomputer connected to a camera and the like as disclosed in U.S. Pat.Nos. 5,640,202 and 5,786,851, for example.

However, actually employed as the frame rate at which a motion pictureis recorded is a predetermined frame rate set according to the system ofa TV monitor and the like or a frame rate set under the control of ahost computer as disclosed in the above US patents, and it cannot beattained that a user manipulating a video camera can optionally changethe frame rate.

As described above, in the conventional technologies, since theresolution of a photographed image is determined according to a presetimage size. Further, since the number of frames to be processed persecond, and the like are uniquely determined by the read-out time of animage pickup element, the processing time of an image compression unit,and the data transfer rate of a recording unit and the like, in theconventional system, a resolution and a frame rate can not be selectedon a camera side.

SUMMARY OF THE INVENTION

An object of the present invention is to solve the above-describedproblems.

Another object of the present invention is to provide an imagegenerating apparatus and an image generating method capable ofgenerating an image, in particular, a motion picture with an optionalresolution or at an optional frame rate.

As a preferred embodiment for such objects, an image generatingapparatus of the present invention includes image pickup means forpicking up an object image at predetermined time intervals andgenerating image pickup data, encoding means for encoding the imagepickup data as motion picture image data having an image rate which isbased on the predetermined time intervals, output means for outputtingthe encoded motion picture image data, and control means for controllingthe output means according to the number of pixels of the image pickupdata picked up by the image pickup means and the image rate.

Further, an image generating method of the present invention includesthe steps of picking up an object image at predetermined time intervalsand generating image pickup data, encoding the image pickup data asmotion picture image data having an image rate which is based on thepredetermined time intervals, outputting the encoded motion pictureimage data, and controlling the output step according to the number ofpixels of the image pickup data picked up in the image pickup step andthe image rate.

Still other objects of the present invention, and the advantagesthereof, will become fully apparent from the following detaileddescription of the embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a digital video camera 100 as an embodimentof an image generating apparatus of the present invention;

FIG. 2 explains read-out executed by a CMOS image pickup element;

FIG. 3 is a block diagram explaining a camera unit 10′ to which thepresent invention is applied;

FIGS. 4A and 4B explain read-out executed by a CCD image pickup element;

FIGS. 5A and 5B are timing charts of the read-out executed by the CCDimage pickup element;

FIG. 6 is a block diagram explaining an encoding unit 40 to which thepresent invention is applied;

FIG. 7 explains the relationship between the number of pixels and aframe processing time;

FIG. 8 explains a sub-band of JPEG 2000;

FIG. 9 is a table showing the relationship between the number of pixelsand a frame processing time; and

FIG. 10 which is composed of FIGS. 10A and 10B is a flowchart explainingthe operation of the digital video camera 100.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The preferred embodiment of the present invention will now be describedin detail hereinafter with reference to the accompanying drawings.

FIG. 1 shows a block diagram of the overall system of a digital videocamera 100 as an example of an image generating apparatus of the presentinvention.

In FIG. 1, reference numeral 1 denotes an image pickup element forconverting optical information into an electric signal, 2 denotes aread-out processing unit for reading out an image signal from the imagepickup element 1, 3 denotes an ACG and A/D conversion unit forsubjecting the read out image signal to gain correction and thenconverting the image signal into digital data, 4 denotes a correctionconversion unit 4 for subjecting the read out digital data to correctionprocessing such as γ correction and the like and converting the digitaldata into image data in a luminance color difference form (Y,Cr,Cb), 5denotes an effect synthesization unit for subjecting the image dataprocessed by the correction conversion unit 4 to arithmetic operation sothat the image data is added with an image effect, 6 denotes a workmemory composed of a RAM and the like for temporarily storing theprocessed data in the respective units 3 to 5, and 10 denotes a cameraunit composed of the respective units 1 to 6 and a ROM 7 which will bedescribed later. Note that the ROM 7 stores information as to the imageprocessing capability (photographing capability) of the camera unit 10.

Further, reference numeral 30 denotes an encoding processing unit forsubjecting the image data input thereto to compression-encodingprocessing, and 31 denotes a work memory composed of a RAM and the likeinto which the encoding processing unit 30 temporarily stores the databeing processed. Note that the encoding processing unit 30compression-encodes an image using the JPEG 2000 system (includingMotion-JPEG 2000) that is an image compression format. Further,reference numeral 40 denotes an encoding unit composed of the encodingprocessing unit 30, the work memory 31 and the respective blocks of aROM 32 which will be described later. Note that the ROM 32 storesinformation as to the image encoding capability of the encodingprocessing unit 30.

Further, reference numeral 50 denotes an output unit for outputting thecompressed image data subjected to the compression-encoding processingin the encoding processing unit 30, 51 denotes a recording unit forrecording the compressed image data output from the recording unit 51,and 52 denotes an external input output terminal for inputting andoutputting the compressed image data output from the output unit 50,other necessary control information, and the like from and to theoutside. An external recording unit, monitor, personal computer (PC),server, and the like are connected to the external input output terminal52. Reference numeral 53 denotes a state detection unit for detectingthe state of the recording unit 51 and the state of the externalrecording unit and the like connected to the external input outputterminal 52.

Reference numeral 60 denotes a control unit composed of a micro-computerand the like for controlling the respective units of the digital videocamera 100, and 61 denotes an operation unit for inputting variouscommands. The control unit 60 appropriately controls the operations ofthe respective units according to a user command input from theoperation unit 61, the information from the respective ROMs, theinformation from the state detection unit 53, and the like.

Next, the operation of the digital video camera 100 shown in FIG. 1 willbe explained in detail with reference to FIG. 1. The processing flow ofimage data to be generated will be explained with reference to FIG. 1.

First, the image of an object is converged on the image pickup element 1through a lens (not shown) and converted into an electric signal. Atthis time, each of the inputs from all the image pickup elements(sensors) is converted into an electric signal of each pixel. Next, theelectric signals are read out by the read-out processing unit 2according to a control signal from the control unit 60 which designatesthe number of pixels (P) and a read-out time (T), thereby generatingimage signals.

Next, the image signals are subjected to gain adjustment and convertedinto digital data in the ACG and A/D conversion unit 3. The digital datais stored once in the work memory 6.

When a predetermined amount of data (for example, an amount of data ofan image of one frame) is stored in the work memory 6, the correctionconversion unit 4 starts to correct and convert the digital data storedin the work memory 6. The correction conversion unit 4 sequentiallyreads out the digital data from the work memory 6, subjects it tophotovoltage-conversion correction (γ correction), further subjects thedigital data to image data conversion into the luminance and colordifference format (Y,Cb, Cr), and outputs the digital data at timingbased on the read-out time of the read-out processing unit 2. Thedigital data is temporarily stored in the work memory 6 as needed, afterthe digital data is converted in the correction conversion unit 4.

The effect synthesization unit 5 starts an operation based on outputtiming of the correction conversion unit 4. The effect synthesizationunit 5 is a section for providing the image data (Y, Cb, Cr data)fetched from the work memory 6 with better-looking by subjecting it to adigital arithmetic operation. When the processing in the correctionconversion unit 4 is completed, the effect synthesization unit 5 readsout the YCbCr data from the work memory 6, subjects the data to digitaleffect processing for making the color of an overall image to sepia bychanging the ratio of Y, Cb, Cr or subjects it to an arithmeticoperation processing for providing a scene change portion with a wipeeffect by synthesizing the digital data with a previous image, andthereafter transfers the image data to the encoding unit 40. Note thatthe processing in the effect synthesization unit 5 may be optionallyexecuted when it is selected or set by the user. When the processing inthe effect synthesization unit 5 is not executed, the image dataprocessed by the correction conversion unit 4 is read out from the workmemory 6 and transferred to the encoding unit 40 bypassing theprocessing in the effect synthesization unit 5.

The encoding processing unit 30 in the encoding unit 40compression-encodes the image data received thereby according to controlinformation which is supplied from the control unit 60 to designate aresolution and a frame processing time. Although an intermediate codegenerated at a processing step is temporarily stored in the work memory31, the image data, which has been finally compression-encoded, istransferred to the output unit 50 and recorded in the recording unit 51or transferred to the external unit from the external input outputterminal 52.

Further, the state detection unit 53 is a block for detecting andmonitoring the recording state of the recording unit 51 or thereceiving/recording state of the external unit connected to the externalinput output terminal 52. A detected state signal is transferred to thecontrol unit 60 as needed and used to control the respective units.

Next, the operations characteristic to the present invention will beexplained below.

First, in the digital video camera 100 of this embodiment, a “resolutionpriority mode” and a “high rate fetching priority mode” can be set as aphotographing mode using the operation unit 61. When the user designatesany one of the resolution priority mode and the high rate fetchingpriority mode, the indicated information is transmitted from theoperation unit 61 to the control unit 60.

Then, the control unit 60 reads the performance of the camera unit 10from the ROM 7 which stores the information of the capability of thecamera unit 10. The performance of the camera unit 10 described here isshown by the information of a maximum number of pixels (Pmax) that canbe read out, a read-out time (Tmax) when the maximum number of pixels(Pmax) is read out, a read-out time (Tmin) when reading-out is executedat a high rate, and the number of pixels (Pmin) that can be read out inthe high rate reading-out.

A method of reading out image data employed when a CMOS sensor is usedas the image pickup element 1 will be explained here using FIGS. 2 and3.

FIG. 2 explains the arrangement of pixels in an image pickup element,FIG. 3 is a block diagram of a camera unit 10′ showing a portion of thearrangement of the camera unit 10 of FIG. 1 in detail. First, a methodof reading out the image data will be explained using the block diagramof FIG. 3.

The camera unit 10′ is composed of a CMOS sensor unit 1′ serving as theimage pickup element, a read-out unit 11, an addition unit 12, and anaddress generation unit 15 serving as the detailed arrangement of theread-out processing unit 2, an AGC unit 13 and an A/D conversion unit 14serving as the detailed arrangement of the ACG and A/D conversion unit3, a pixel designation unit 16, a read-out rate designation unit 17serving as the detailed arrangement of the control unit 60, a controlunit 60′ composed of other control systems, and further the correctionconversion unit 4.

The operation of the camera unit 10′ of FIG. 3 will be explained.

When the number of pixels is designated by the pixel designation unit 16under the control of the control unit 60′, information for designatingthe pixels to be read out is transferred to the address generation unit15.

If it is designated here to read out the maximum number of pixels thatcan be read out, the address generation unit 15 generates an address,which causes electric signals to be sequentially read out from all thepixels of the CMOS sensor 1′, to the read-out unit 11. At this time, inFIG. 2, the electric signals of all the pixels as shown by Pi, j; Pi,j+1; . . . ; Pi+1, j; Pi+1, j+1; . . . are sequentially read out. Inthis case, since all the pixels are read out, the maximum time Tmax isnecessary to read one image.

Further, if it is designated to read out one fourth the maximum numberof pixels, the address generation unit 15 generates an address forreading out the pixels of the CMOS sensor 1′ in a unit of four pixels tothe read-out unit 11. At this time, in FIG. 2, one signal is read outfrom Pi, j; Pi, j+1; Pi+1, j; Pi+1, j+1 according to a rule determinedthereto so that one pixel is read out from adjacent four pixels (forexample, only Pi, j of Pi, j; Pi, j+1; Pi+1, j; Pi+1, j+1 are read).That is, since all the pixels of the one image is read out in the unitof four pixels, they can be read in a read-out time that is one fourththe maximum time Tmax.

As described above, the read-out unit 11 reads out the electric signalsbased on the designated number of pixels, and the read-out electricsignals are added by the addition unit 12, subjected to gain adjustmentin the AGC unit 13, and converted into the digital data in the A/Dconversion unit 14. Since the electric signals are added in the additionunit 12, this provides the same result as that of a case in which theelectric signals are processed by a half-band low-pass filter, therebyeliminating influence of aliasing distortion, which is caused in offsetsampling of.

As described above, since a smaller number of pixels (P) to be read outdecreases the read-out time (T) of one image, the value of the ratedesignated by the read-out rate designation unit 17 is changed under thecontrol of the control unit 60. That is, the read-out rate designationunit 17 generates a timing signal corresponding to a present read-outtime of one image. The A/D conversion unit 14 executes processing inresponse to the timing signal as well as notifies a processing starttiming to the correction conversion unit 4.

As described above, since the number of pixels (P) and the read-out time(rate) (T) approximately have the relationship shown by the followingexpression (1).

P×T=constant  (1)

Accordingly, when P or T is set by the user, T or P paired with eachother can be determined from the maximum capability of the image pickupelement.

For example, referring to FIG. 9 to explain the number of pixels (P) andthe rate (T) of an image pickup element having 3,000,000 pixels, therate is set to 30 frames/sec in an ordinary motion picture (VGA size).When, however, 4 pixels are read out together as described above, sincethe number of pixels is reduced to ¼, it is possible to read out thepixels at a rate of 120 frames/sec. Conversely, when an image pickupelement has the number of pixels twice as many as that of VGA, an imagehaving a maximum number of 1,300,000 pixels (SXVGA) can be read out at arate of 7.5 frames/sec.

As described above, since the CMOS sensor has the maximum read-out pixel(Pmax) and the maximum read-out time (Tmax) as its performance, the usercan optionally select in this system any of them or an intermediatenumber of pixels and an intermediate read-out time.

Next, the reading method employed when the image pickup element 1 iscomposed of a CCD image pickup element will be explained with referenceto FIGS. 4A and 4B and FIGS. 5A and 5B.

An operation for reading out an ordinary overall image will be explainedwith reference to FIG. 4A.

The charges accumulated in the CCD image pickup element are verticallytransferred through a CCD light receiving unit 20 serving as an imagepickup element surface at timing generated by a timing generator (TG)22. The charges transferred vertically are transferred to the ACG andA/D conversion unit 3 in a line unit by a horizontal transfer unit 21.As described above, the CCD image pickup element employs such a transfersystem that all the pixels are vertically transferred and thentransferred horizontally. Note that the CCD light receiving unit 20serves as a read-out area in its entirety.

FIG. 5A shows in time-series a vertical synchronous signal (VD),horizontal synchronous signal (HD), a pixel reading signal (PD), clocksignal (Clk) acting as the reference signal thereof, an operation signalin A/D conversion capable of reading A/D conversion data at 3 Clk, andtiming (RD) at which the A/D conversion data is read.

As shown in FIG. 4A, the pixel (m×n) data of the light receiving unit 20is read at the timing PD that is in synchronism with the verticalsynchronous signal VD and the horizontal synchronous signal HD shown inFIG. 5, and subjected to A/D conversion at the timing RD. Note that FIG.7 shows an image in which the data of all the images 102 is read at thetiming of the vertical synchronous signal VD.

Next, a case in which the CCD image pickup element reads out data havinga different resolution (a different number of pixels) is explained withreference to FIG. 4B. In FIG. 4B, when a read-out area (m′×n′) 23 iscomposed of an area including the number of pixels designated by thepixel designation unit 16, in the CCD light receiving unit 20, a methodof reading out the data of the read-out area (m′×n′) 23 will beexplained below. Note that FIG. 5B shows timing at which processing isexecuted in time-series.

In FIG. 4B, first, when pixels are transferred in a vertical direction,the pixels located under the read-out area 23 (pixels of k lines in avertical pixel direction) are skipped at a high rate in synchronism withthe clock signal Clk (PD 1, 2, . . . , k of in FIG. 5B). Then, when linedata (k+1) including the data in the read-out area 23 is transferred tothe ACG and A/D conversion unit 3 by the horizontal transfer unit 21,initial j pixels are skipped at a high rate (Clk 1, 2, . . . , i in FIG.5B), and pixels j+1 to j+n′ in the read-out area 23 are subjected to A/Dconversion processing in synchronism with the clock Clk (PD 1, 2, . . ., n′ of FIG. 5B). The remaining i pixels are skipped at a high ratesimilarly to the j pixels (Clk 1, . . . , i of FIG. 5B). The aboveoperations are repeated for m′ lines (PD n′+1, n′+2, . . . , m′n′ andClk 1, . . . , i in FIG. 5B), and the pixels of the remaining h linesare skipped vertically at the high rate similarly to the k lines (PD 1,2, . . . , h in FIG. 5B). Note that FIG. 7 shows an image in which thedata of a pixel area 101 which is a part of the data of all the images102 is read by the processing described above at timing of the verticalsynchronous signal VD which is shorter than the timing at which all thepixels are fetched.

As described above, in the CDD image pickup element, aliasing distortiondue to sampling is not caused because pixels are partly read out inplace of thinning-out reading. In contrast, the portions skipped at thehigh rate require a somewhat overhead processing time. However, sincethe pixel data other than that in the effective the read-out area 23 istransferred at the high rate also in the CCD image pickup element, if asmall number of pixels (P=n×m) is read out, one image can be read out ina short time (T), thereby the expression (1) can be establishedsimilarly to the CMOS image pickup element.

Further, one of the features of the partial reading executed by the CCDimage pickup element resides in that the pixels (n×m) to be clipped canbe relatively optionally set differently from the CCD image pickupelement in which a resolution is changed by four times each.Accordingly, it is possible to construct a camera system capable ofrelatively optionally select the number of pixels as shown in FIG. 9.

From what has been described above, the user can select a photographingmode so that the number of pixels and the rate have a complementaryrelationship in any of the CMOS and CCD image pickup elements. Further,an image can be read out so that the image pickup element can exhibitits maximum capability.

Next, an arrangement for compressing and encoding the image data readout from the image pickup element in the encoding unit 40 in FIG. 1 asdescribed above will be specifically explained with reference to FIG. 6.

In FIG. 6, the encoding unit 40 is composed of the work memory 31, adiscrete wavelet transform circuit 33, a quantization circuit 34, anentropy coding circuit 35, a code stream generation circuit 36, and apacket control circuit 37 as its detailed arrangement. Note that theabove units 33 to 37 are processing blocks for constituting the encodingprocessing unit 30 described above.

First, the frame processing time and the number of processing pixels inthe encoding unit 40 are set such that an operation is executed based onthe number of pixels (P) and the read-out time (T) determined by thecontrol unit 60 according to whether the user sets the resolutionpriority mode or the high rate capture mode described above.

This embodiment assumes that the encoding unit 40 can deal with theperformance of the camera unit 10, i.e. the maximum number of read-outpixels (Pmax) and the shortest frame read-out time (Tmin). That is, theencoding unit 40 is equipped with the size of the work memory 31 and thearithmetic operation capabilities in the respective blocks, which areset such that a frame image having the number of pixels (P) and theframe read-out time (T), which are uniquely set by the expression (1)according to the performance of the camera unit 10, can be processed atreal time.

The discrete wavelet transform circuit 33 subjects the input image datato sub-band coding according to the set number of pixels and processingtime. According to the discrete wavelet transform circuit 33 employingthe JPEG 2000 system used here, the image data is two-dimensionallysub-band coded to L- and H-components in horizontal and verticaldirections and converted into LL, LH, HL, and HH sub-band coefficients.The LL component of these components is subjected to two-dimensionaldiscrete wavelet transform again, thereby 2LL, 2LH, 2HL, and 2HHsub-band coefficients are obtained. Further, when recursive processingfor subjecting the 2LL component to the two-dimensional discrete wavelettransform again is executed a predetermined number of times, importantdata is concentrated to the LL component and a noise component isconcentrated to the HH component. When the image data is adaptivelyquantized by the quantization circuit 34 making use of the abovecharacteristics, the image data can be compressed.

FIG. 8 conceptually shows the relationship between the sub-bandcoefficients and the image data when the two-dimensional discretewavelet transform is executed three times. As described above, in thediscrete wavelet transform of the JPEG 2000 system, the data encoded asto the resolution is encoded in the scalability structure of theresolution in order to code the image data recursively. Specifically,encoded data having the number of pixels (resolution) reduced by onefourth each is generated as shown below.

all the images>1LL>2LL, . . . >nLL

where, (n−1)LL=nLL+nHL+nLH+nHH

Next, the entropy coding circuit 35 further compresses theintermediately encoded image data subjected to the discrete wavelettransform. Although the entropy coding is not described here in detail,it breaks down quantized data to bit plains in the unit of adjacentpixel blocks (code blocks), and the binary data of each bit plain isarithmetic-encoded based on a context model. The data encoded using therespective bit plains contributes to improve the quality of a reproducedimage being transmitted, by optimally setting an encoding order. Thatis, the encoded data of the respective code blocks is divided intolayers having a correlation with the quality of a reproduced image beingtransmitted.

The data encoded as described above is minutely divided into the layersof the code blocks, and a code stream is generated by arranging theencoded data as packets based on a predetermined character. Thecharacter arranges the encoded data, which is located at similarpositions and has similar image quality, as a packet. The packets of thecode stream are generated by the code stream generation circuit 36.

Further, when the JPEG 2000 system is used, some characteristicfunctions can be realized by variously arranging the packets of the codestream. For example, when the packets are preferentially arranged in theorder of higher layer (in the order of good SN ratio), the code streamis sequentially decoded such that rough-quality image (image having abad SN ratio) is gradually made to a smooth image (image having a goodSN ratio). Further, when the packets are preferentially arranged in theorder of the hierarchy of the sub-bands of the discrete wavelettransform described above, the code stream is sequentially decoded suchthat a blurred image (image having a bad resolution) is made to a clearimage (image having a good resolution). The scalability functions asdescribed above are called an SN scalability and a resolutionscalability.

In this embodiment, the scalabilities are changed by the code streamgeneration circuit 36 according to the number of pixels and the rate setby the control unit 60. Specifically, in a high resolution image (highpixel-density image) in which the frame rate of a motion picture is 30frames/sec or less, the packets of the code stream are arranged in theorder of the SN scalability and output to the packet control circuit 37,and in a high rate image having a frame rate exceeding 30 frames/sec,the packets of the code stream are arranged in the order of theresolution scalability and output to the packet control circuit 37.

The packet control circuit 37 is a circuit for controlling the mode ofthe code stream generated as described above, before it is output to theoutput unit 50 at a rear stage. Although the packet control circuit 37ordinarily outputs the generated code stream as it is, when controlsignal according to a recording state is output from the control unit60, the packet control circuit 37 outputs the code stream according tothe control information. The arrangement for outputting the code streamaccording to the control information will be explained below.

As shown in FIG. 1, the digital video camera 100 of this embodimentincludes the state detection unit 53 for detecting and monitoring therecording state or the receiving state of the recording unit 51 or theexternal unit connected to the external input output terminal 52, andthe control unit 60 for controlling the operations of the respectiveunits according to the state information detected therein. For example,when the recording unit 51 is busy, even if the code stream generated bythe encoding unit 40 is output from the output unit 50, it cannot berecorded. Thus, when the recording state detection unit 53 detects thebusy state of the recording unit 51, it notifies the busy state to thecontrol unit 60 as information. When the control unit 60 receives thisstate information, the packet control circuit 37 stops transmission ofthe packets of the code stream in a frame unit and performs control suchthat a predetermined frame image finish sequence is executed. That is,when the recording unit 51 is busy, the control unit 60 performs controlsuch that all the encoded data of the respective frame is not output,while only a predetermined number of packets are output and transfer ofthe packets exceeding the predetermined number of the packets isstopped.

The same control can be performed to the external recording unitconnected to the external input output terminal 52 and to the displayunit likewise, as well as the recording unit 51. When the recordingstate detection unit 53 receives the status of the external unit, suchas the receiving/recording states and the like, through the externalinput output terminal 52 by means of a command communication and outputsthe state information to the control unit 60, the control unit 60 cancontrol the output of the packet control circuit 37 according to thestate of the external unit.

Since the scalability functions of the JPEG 2000 are set in the order ofthe packets, even if the encoded data of the respective frames iscancelled halfway, an image having appropriate quality can be decoded.Specifically, since the SN scalability is employed in a motion picturehaving high image quality and a low rate, even if the encoded data ofthe respective frames of the image is cancelled halfway, it is possibleto reproduce an image having an optimum image quality and thedeterioration of the quality of the image can be minimized. In contrast,since the resolution scalability is employed in a motion picture havinga low pixel-density and high rate, even if the encoded data of therespective frames of the image is cancelled halfway, it is possible touniformly reproduce an entire image, thereby the high rate motion of anobject, which can be hardly predicted, can be fetched.

Note that, in this embodiment, header information (the headerinformation of a motion JPEG 2000 file format, and the like), which isnecessary in filing processing executed after the code stream arerecorded in the recording unit 51, as well as the cancel of transmissionof the packets, is stored and held in the memory region of the controlunit 60. On the completion of recording of the stream information, theheader information is transferred to the recording unit 51 and a fileformat is generated, thereby the recording is finished. The aboveprocessing is executed likewise when the header information is output tothe external recording unit.

Note that although the present invention describes that the file formatis generated to the recording unit 51 after a series of the motionpicture recording operation is finished, it is also possible to writethe file header at every predetermined time making use of the moof Boxfunction of the Motion JPEG 2000.

Although what has been described above is processing executed assumingthat the encoding unit 40 can deal with the maximum number of red-outpixels (Pmax) and the shortest frame read-out time (Tmin), processingexecuted in other cases will be explained below.

Here, processing will be described which is executed when theperformance of the encoding unit 40 is inferior to that of the cameraunit 10. Specifically, this case assumes that the encoding unit 40 isnot equipped with the size of the work memory 31 and an arithmeticoperation processing capabilities capable of processing at real time theframe image having the number of pixels (P) and the frame read-out time(T), which are uniquely set by the expression (1) according to theperformance of the camera unit 10.

The operation in the above case will be explained with reference to FIG.1.

First, when the user sets any one of the resolution priority mode andthe high rate fetching priority mode, the control unit 60 receives thisinformation and reads out the performance information (Pmax, Tmax) ofthe camera unit 10 from the ROM 7. The control unit 60 further reads outthe encoding performance information (P′max, T′min) of encoding unit 40from the ROM 32. The encoding performance information is represented bythe processing rates of the respective blocks in the encoding unit 40,the capacity of the work memory 31, and the like.

Here, when the photographing mode designated by the user is theresolution priority mode, P shown by the following expression (2) isselected.

P=Min(Pmax,P′max)  (2)

That is, the largest value of the number of pixels that can be processedby both the encoding unit 40 and the camera unit 10 is selected. Then,the processing rate T is calculated from the expression (1), and thenumber of pixels and the rate are set to the camera unit 10 and theencoding unit 40. Subsequent operations are the same as those of therespective blocks described above.

In contrast, when the user selects the high rate fetching priority mode,the processing rate T is selected from the following expression (3).

T=Max(Tmin,T′min)  (3)

That is, the smallest value of the processing rates that can beprocessed by both the encoding unit 40 and the camera unit 10 isselected. Then, the number of pixels P is selected from the expression(1), and the number of pixels and the processing rate are set to thecamera unit 10 and the encoding unit 40. Subsequent operations are thesame as those of the respective blocks described above.

As explained above, since the processing capabilities of the camera unit10 and the encoding unit 40 are separately stored and held in the ROM inthis embodiment, when there is other compatible camera unit, it ispossible to exhibit a maximum capability as a system by comparing itsperformance in the same manner.

Further, in the system of this embodiment, since the camera unit 10 canexecute partial read-out, and the like, the user can select thephotographing mode. However, when other high pixel-density camera unit,which cannot executed the partial read-out, and the like, is combinedwith the encoding unit 40 described above, the code stream isautomatically generated in the resolution priority mode and when a lowpixel-density camera unit, which can execute read-out at a high rate, iscombined with the encoding unit 40 inversely, the code stream isautomatically generated in the high rate fetching priority mode, thesystem can generate the code stream in an optimum mode without the needof selection made by the user.

The operation described above will be explained with reference to theflowchart of FIG. 10. That is, FIG. 10 shows an operation flowchart ofthe digital video camera 100.

In FIG. 10, first, the control unit 60 determines whether or not thecamera unit 10 can partially read out P pixels from all the pixels andwhether or not the read-out time (T) of one frame (P pixels) is variable(step S1000).

When the determination at step S1000 is “variable”, i.e. when the numberof pixels (P) to be read out and the frame rate (T) are variable, thecamera performance information (Pmax, Tmax) is read out from the ROM 7as well as the encoding performance information (P′max, T′min) is readfrom the ROM 32 (step S1010).

Next, the control unit 60 determines whether or not the photographingmode designated by the user using the operation unit 61 is theresolution priority mode due to high pixel-density or the high ratefetching priority mode (step S1020).

When it is determined by the control unit 60 that the “resolutionpriority mode” is designated as the photographing mode, the maximumnumber of pixels, which can be processed by both the camera unit 10 andthe encoding unit 40, is determined from the expression (2), and therate (T) corresponding to the number of pixels (P) is calculated fromthe expression (1) (step S1030).

In contrast, when it is determined by the control unit 60 that thephotographing mode is designated to the “high rate fetching prioritymode”, the minimum processing time, which can be processed by both thecamera unit 10 and the encoding unit 40, is determined from theexpression (3), and the number of pixels (P) corresponding to the frameprocessing time (T) is calculated from the expression (1) (step S1032).

Next, the numbers of pixels (P) and processing times (T) determined bythe control unit 60 as described above are set to the camera unit 10 andthe encoding unit 40 (step S1040).

The above processing is executed in the camera unit in which the numberof pixels (P) and the frame processing time (T) are variable. When,however, it is determined at step S1000 that the number of pixels (P)and the frame processing time (T) are “fixed”, i.e. in the camera unit10 in which any one of the number of pixels (P) and the frame processingtime (T) is fixed, the control unit 60 detects the performance of thecamera unit 10 (step S1002). That is, the control unit 60 automaticallydetermines the number of pixels (P) and the frame processing time (T) bydetecting the number of fetched pixel and processing time the cameraunit 10.

In this case, the processing, in which the user designates thephotographing mode, is not employed, and the control unit 60 sets aparameter for automatically and uniquely determining a high resolutionmode or a high rate fetching mode. Specifically, the frame processingtime (T), which serves as a determination criterion of post processingthat is changed depending on the high resolution mode and the high ratefetching mode, is determined, and when the control unit 60 determinesthat the camera unit 10 gives priority to resolution, the frameprocessing time (T) is set to 1/30 sec or more, and when the controlunit 60 determines that the camera unit 10 can reads out data at a highrate although having a low pixel density, the frame processing time (T)is set to less than 1/30 sec. With the above arrangement, the postprocessing can be changed automatically.

Next, the control unit 60 sets the number of pixels (P) and the frameprocessing time (T), which are automatically set from the camera unit10, to the encoding unit 40 (step S1004), thereby the setting processingin the camera unit 10 is completed.

As described above, although the number of pixels (P) and the frameprocessing time (T) are set by somewhat different processing dependingon whether the performance of the camera unit is variable or fixed, thecamera unit 10, to which the number of pixels (P) and the frameprocessing time (T) are set as described above, reads out a frame imageusing predetermined the number of pixels (P) and the frame processingtime (T), and the encoding unit 40 executes image compression processing(step S1050).

Here, at the final step of the image compression processing, the packetsare rearranged as described below by the determination of the controlunit 60. First, the control unit 60 determines the frame processing time(T) (step S1060). When the determination at step S1060 is “No”, i.e.when it is determined that frame processing time (T) is 1/30 sec ormore, the code stream is generated such that the packets of the JPEG2000 are arranged in correspondence to the SN scalability (step S1070).In contrast, when the determination at step S1060 is “Yes”, i.e. when itis determined that frame processing time (T) is less than 1/30 sec, thecode stream is generated such that the packets of the JPEG 2000 arearranged in correspondence to the resolution scalability (step S1072).

Next, the control unit 60 determines the state of recording unit 51 orthe external unit connected to the external input output terminal 52,which receive and record the code stream generated by the aboveprocesses, according to the information of the state detected by thestate detection unit 53 (step S1080). Specifically, the control unit 60determines whether or not the recording unit 51 or the external unit isin a “busy” state, and as examples of the busy state, there arecontemplated whether or not a recording operation overflows, whether ornot a transmission path is busy, and whether or not various data can bereceived. In the example of the digital video camera (or digital stillcamera) on which the image generating apparatus of the present inventionand the recording unit 51 are mounted, it is important to detect whetheror not the recording unit 51 overflows the recording rate.

When the it is determined that the state is “normal” at step S1080, i.e.when it is determined that the receiving/recording state of therecording unit 51 is normal, the code stream is continuously output asit is (step S1090), and the receiving state is continuously monitoreduntil the frame is completed (step S1100), and then the process isrepeated from step S1080.

In contrast, when it is determined at step S1080 that thereceiving/recording state is “busy”, the output of the code stream isstopped once until the code stream of a next frame is output (stepS1092). Then, when the next frame is started, the code stream generationchange operation is repeated from step S1050 as long as it is notdetected that the processing of the camera unit 10 is ended (stepS1110).

When it is determined at step S1110 that the processing executed by thecamera unit 10 is ended, the control unit 60 transfers the headerinformation and the like and completes the recording operation, and thenexecutes processing for generating a file format (step S1120), therebythe operation of this flow is ended.

The flowchart of FIGS. 10A and 10B have been described above.

Since the processing executed by the camera unit 10 and the encodingunit 40 is changed between the resolution priority mode and the highrate fetching priority mode as described above, the code streamgenerated in the resolution priority mode can maintain image quality ashigh as possible in correspondence to the resolution priority mode aswell as the code stream generated in the high rate fetching prioritymode can fetch the instant motion of an object in it entirety as much aspossible in correspondence to the high rate fetching priority mode.

It should be noted that although the recording unit 51 has been mainlyexplained as the unit to which the code stream is output in thisembodiment, the same arrangement can be realized even if the externalinput output terminal 52 is exemplified as the unit to which the codestream is output. Further, even if the unit connected to the externalinput output terminal 52 is a network such as the Internet, the sameeffect can be obtained by controlling output of the code stream bydetecting the busy state of the network by a TCP/IP protocol.

Further, even if the camera unit 10 and the encoding unit 40, whichconstitute the image generating apparatus of the embodiment, areintegrated together with the recording unit 51 as in the digital videocamera and the digital still camera, and even if the camera unit 10, theencoding unit 40, and the recording unit 51 are entirely or partlyseparated from each other, the present invention can be realizedsimilarly.

Note that the recording unit 51 includes a disk-like recording mediumsuch as a Blu-ray, DVD, and the like, a memory card such as an SD card,compact flash and the like, a recording medium such as a hard disc,tape, and the like and the recording units thereof. Further, therecording unit 51 may have a replay function.

Further, it is needless to say that the object of the present inventioncan be also achieved by supplying a recording medium, in which theprogram codes of software for realizing the functions of the embodimentof the present invention as described above are recorded, to a system oran apparatus and by reading and executing the program codes stored inthe recording medium by the computer (or CPU or MPU) of the system orthe apparatus.

In this case, the functions of the embodiment described above isrealized by the program codes themselves read out from the recordingmedium, and the recording medium storing the program codes constitutesthe present invention.

Further, it is needless to say that the case, in which not only thefunctions of the embodiment described above are realized by executingthe program codes read out by the computer but also an operating system(OS) operating on the computer partly or entirely executes actualprocessing steps in response to the indications from the program code sothat the functions of the embodiment described above are realized, isalso included in the present invention.

As described above, according to the present invention, it is possibleto generate the stream of a motion picture which makes full use of thecharacteristics of an image such as the high resolution and the highrate fetching, by changing the output of the stream according to theintervals at which an image frame are processed, and the number ofpixels of the image frame.

Many widely different embodiments of the present invention may beconstructed without departing from the spirit and scope of the presentinvention. It should be understood that the present invention is notlimited to the specific embodiments described in the specification,except as defined in the appended claims.

1-17. (canceled)
 18. An image data generating apparatus comprising: adesignation unit configured to designate the number of pixels of animage to be input; an image pickup unit configured to pick up the imagecorresponding to the designated number of pixels at a predetermined timeinterval and to generate image pickup data of the picked-up image; anencoding unit configured to encode the image pickup data as motionpicture image data having an image rate which is based on thepredetermined time interval; an output unit configured to output themotion picture image data encoded by said encoding unit; and a settingunit configured to set scalability for the motion picture image data tobe output by said output unit, in accordance with the designated numberof pixels designated by said designation unit and the image rate of themotion picture image data.
 19. An apparatus according to claim 18,wherein said output unit changes an arrangement order of data packetsincluded in the motion picture image data, in accordance with thescalability set by said setting unit.
 20. An apparatus according toclaim 18, wherein said setting unit effects the setting so as to outputthe motion picture image data in accordance with an SN scalability ifthe image rate is less than or equal to a predetermined rate, andeffects the setting so as to output the motion picture image data inaccordance with a resolution scalability if the image rate is greaterthan the predetermined rate.
 21. An apparatus according to claim 18,wherein said encoding unit encodes the image pickup data using asub-band encoding method, and said output unit selectively executes aresolution scalability output mode and an SN scalability output mode asan output mode of the motion picture image data.
 22. An apparatusaccording to claim 18, further comprising: a first memory unitconfigured to store information on the image pickup capability of saidimage pickup unit; a second memory unit configured to store informationon the encoding capability of said encoding unit; and a calculation unitconfigured to calculate at least one of the designated number of pixelsand the image rate of the motion picture image data by using theinformation stored in said first memory unit and the information storedin said second memory unit.
 23. An image data generating methodcomprising the steps of: designating the number of pixels of an image tobe input; picking up the image corresponding to the designated number ofpixels at a predetermined time interval and generating image pickup dataof the picked-up image; encoding the image pickup data as motion pictureimage data having an image rate which is based on the predetermined timeinterval; outputting the motion picture image data encoded in saidencoding step; and setting scalability for the motion picture image datato be output in said outputting step, in accordance with the designatednumber of pixels designated in said designating step and the image rateof the motion picture image data.